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Purpose: Describe the relationship between medical students' self-assessment and goal-setting (SAGS) skills 
and development of interviewing skills during the first-year doctoring course. 

Method: 157 first-year medical students completed three two-case standardized patient (SP) interviews. After 
each of the first two, students viewed videotapes of their interview, completed a SAGS worksheet, and 
reviewed a selected tape segment in a seminar. SAGS was categorized into good and poor quality and 
interviewing skills were rated by trained raters. 

Results: SAGS improved over time (37% good week 1 vs. 61% good week 10). Baseline SAGS and 
interviewing skills were not associated. Initial SAGS quality was associated with change in interviewing 
skills - those with poor-quality SAGS demonstrated a decrease and those with good-quality SAGS 
demonstrated an increase in scores by 17 weeks (ANOVA _F = 4.16, p =0.024). For students whose SAGS 
skills were good at both week 1 and 10, interviewing skills declined in weeks 1-10 and then increased 
significantly at week 17. For those whose SAGS remained 'poor' in weeks 1-10, interviewing skills declined in 
weeks 10-17. 

Conclusions: In general, the quality of students' SAGS improved over time. Poor baseline SAGS skills and 
failure to improve were associated with a decrease in interviewing skills at 17 weeks. For students with better 
SAGS, interviewing skills increased at week 17. Improvement in SAGS skills was not associated with improved 
interviewing skills. Understanding structured self-assessment skills helps identify student characteristics that 
influence progressive mastery of communication skills and therefore may inform curriculum and remediation 
tailoring. 
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Medical education should ensure that students 
have life-long, self-directed learning skills essen- 
tial for continued professional expertise devel- 
opment after formal training has ended (1). The ability to 
self -monitor and regulate learning through self-assessment 
and setting goals are key components of achievement 
in almost all learning domains (2). The development of 
expertise, as demonstrated through consistent optimal per- 
formance of skills, is attained through cycles of challen- 
ging practice with accurate feedback from experienced 
coaches (3). In order to practice safely, it is crucial that 
physicians develop the capacity and habit to self-monitor 
their cognitive and non-cognitive expertise. This is par- 
ticularly true if medical education is to become more 



flexible and individualized in order to better meet the 
health needs of society (4). 

Effective communication with patients is a critical 
component of a physician's diagnostic accuracy, thera- 
peutic rapport, and ability to educate and counsel patients 
(5, 6). Studies suggest that the medical interview is the 
source of 60-80% of the data needed for diagnosis (7-9). 
As a result of a hardy research base, there is a fundamental 
consensus on mastery criteria for interviewing skills and 
evidence for effective communication curriculum (10-13). 
Both self-assessment and peer-assessment are commonly 
incorporated in medical schools' communication curricula 
and have been shown to enhance communication skills 
as measured by performance on Objective Structured 
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Clinical Exams (OSCE) (14). While the evidence is over- 
whelming that physicians cannot perform summative self- 
assessments accurately, it has been suggested that their 
situation-relevant self-assessment for the purpose of self- 
monitoring can be honed and is critical to ongoing 
learning from experience (15, 16). As Eva and Regehr (17) 
point out 'safe practice requires that self-assessment be 
conceptualized as repeatedly enacted, situationally rele- 
vant assessments of self-efficacy and ongoing "reflection- 
in-practice'" (17). 

There are many reports on the validity and accuracy of 
medical students self-assessment (1, 18) as well as the 
contribution of self-assessment programs to 'promote 
more mature, collegial and productive learning environ- 
ments particularly suited to the training of healthcare 
professionals' (15). 

Chang's cross-sectional study (2011) described content 
of students' learning goals and the relationship of 
students' learning goals to the students' interviewing 
skills in a clinical performance exam (19). However, we 
found no longitudinal studies such as ours examining the 
relationship of medical students' self-assessment and 
goal-setting (SAGS) ability with the development of 
clinical skills over time. To refine clinical skills curricula 
and ensure expertise development, it is important to 
understand the diversity in and role of students' capacity 
for self-assessment, self-monitoring and goal setting in 
the development of communication skills. 

We report on the qualities of students' goal-setting 
based on self-assessment of medical interviewing perfor- 
mance and how SAGS are associated with both base- 
line and subsequent medical interviewing performance 
in the context of a long established interviewing skills 
curriculum for first-year medical students. 

We introduced a sequence of individual and small- 
group videotape reviews (VTR) to our first-year doc- 
toring course. The goal of the VTR sessions was to 
structure students' self-assessed interviewing skills using 
literature-based mastery criteria (as reflected in the 
checklist) and facilitate the development of an effective 
learning plan. We address three core questions in this 
study: 

1) Does participating in a guided videotape re- 
view process of self-assessment and goal setting 
(SAGS) enhance the quality of students' SAGS 
skills? 

2) Is the quality of students' initial SAGS skills asso- 
ciated with concurrent interviewing skills or with 
changes in interviewing skills over the course of a 
17-week, first-year doctoring course? 

3) Is improvement in SAGS skills associated with the 
development of interviewing skills? 



Methods 

Setting and subjects 

All 165 first-year medical students at this Northeastern 
U.S. private medical school are required to successfully 
complete the 50-h Practice of Medicine (POM) course, 
which includes modules on doctor-patient communica- 
tion, physical diagnosis, health policy and preventive 
medicine, culture and diversity, and ethics. A variety of 
instructional formats are employed including lectures, 
small-group seminars (eight students to two faculty), a 
longitudinal clinical preceptorship, standardized patient 
(SP) exercises with faculty and SP feedback and small- 
group videotape review (G-VTR). At the time of this study, 
a structured self-assessed videotape review (S-VTR) pre- 
ceding the G-VTR had been recently added. 

As part of an ongoing IRB-approved Medical Student 
Research Registry, all incoming medical students are 
asked to consent to allow their educational data (routi- 
nely collected as part of their medical school experience) 
to be compiled in a de-identified database and used for 
research purposes; 157 of 165 students provided this 
consent and only their data were used in this study. 

Design 

This prospective cohort study assessed students' inter- 
viewing skills over three points of time in their first-year 
doctoring course (initial, Week 10 and Week 17) and 
assessed their SAGS skills at the first two time points 
(initial and Week 10). 

Standardized patient exercises 

In 2009, POM student class of 2013 participated in three 
SP case exercises scheduled over a four and a half-month 
period. The exercises took place over a 2-h period in 
which each student interacted with each of two SPs for 
10 min and then received 4 min of immediate verbal 
feedback from a faculty observer. In each exercise, at 
least one of the two SP interactions was faculty observed 
and one was videotaped. Student participation was 
required. 

Case content and interviewing skills aligned with 
material covered in the curriculum and were designed 
to be progressively more challenging over time. SP case 
training materials were written by course faculty and 
included a detailed role description. Students were told, 
in general, what case content to prepare for and in some 
cases were given brief primers to help them prepare. 
A previously validated interviewing skills checklist was used 
consistently across all SP interactions to assess student 
skills (20-22). SPs participated in both case portrayal 
training (3 h) and checklist rater training (2 h), both with 
experienced SP trainers. 
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Self-assessment videotape review 

Students were assigned the task of reviewing the video- 
tape of their interview on their own time the week 
following each exercise in preparation for the faculty 
facilitated Group VTR. Students were given a worksheet 
that contained a checklist similar to that used by the SPs 
that provided benchmarks to guide their self-assessment. 
On the same worksheet, students were asked to write brief 
descriptions of their demonstrated strengths and weak- 
nesses, and then identify three goals for improvement and 
a concrete plan for doing so (see Appendix). 

Group videotape review 

After the first and second SP encounter and S-VTR, 
students reviewed a 2-3 min segment of their interview in 
class. During this 2 h, faculty facilitated G-VTR, each of 
the eight students shared their videotape segments. Before 
showing the clip they were expected to state why they 
chose this particular segment and to ask the group for 
focused feedback. The faculty member modeled giving 
specific feedback on modifiable behaviors and encour- 
aged the students to do so for each other. They did not 
give explicit feedback on the students SAGS but simply 
reinforced the benchmarks and modeled identifying 
behaviorally specific feedback and concrete improvement 
plans. 

Rating of quality of self-assessment and goal setting 

We defined quality of students' SAGS by the specificity 
with which the students were able to articulate observa- 
tions about their interview skills and then delineate a 
specific and reasonable plan to remedy the perceived 
deficiencies. A research assistant, blind to interviewing 



performance, rated the quality of the students' written 
SAGS using a three-point scale (poor, fair, good). Table 1 
provides criteria (and examples) for rating the quality. 
A random sample of 31 'worksheets' were re-rated by 
two faculty members and agreement between the original 
rating and faculty member A was 83% and faculty 
member B was 86% and agreement between the faculty 
raters (faculty members A and B) was 87%, suggesting 
high inter-rater reliability. For subsequent analyses, poor 
and fair were combined (given that only 10% of work- 
sheets were rated as 'poor') and a dichotomous rating 
scale of poor/fair versus good was used. 

Rating of interviewing skills 

In all three exercises, SPs assessed the students' inter- 
viewing skills using a behaviorally anchored, 17-item 
communication skills checklist that included items in 
three core communication domains: information gather- 
ing, relationship development, and patient education and 
counseling. Checklist items were rated as not done, partly 
done, and well done (each with associated behavioral 
anchors) and an overall interviewing skills score was 
calculated as the percent of items rated as 'well done'. 
Internal consistency of the checklist was strong with 
Cronbach's alpha ranging from 0.83 to 0.88 across the 
three SP exercises. 

Because the SP exercises were explicitly designed to 
increase in difficulty over the course of the academic year, 
percent of items rated well-done for each exercise were 
transformed into Z-scores (z-scores are expressed as 
standard deviations above or below the mean) to facil- 
itate the comparison of relative improvement in skills 
over time while accounting for increases in case-difficulty. 



Table 1. Rating of the quality of self-assessment and goal setting (SAGS) (n = 129) 

Poor Fair Good 

Initial rating 13(10%) 70(54%) 46(36%) 



Criteria for 
rating 



Examples 



. Major elements 
missing/incomplete 

. Superficial 

. Goals focused on 
simply needing to 
improve 

Self-Assessment: 

I'm too nervous 

Goal Setting: 

Don't stress about 



Dichotomized 



. Completed 

. Broad, not sufficiently detailed 

assessment 
. Goals too broad, not actionable 
. Goals and self-assessment not fully 

aligned 
Self-Assessment: 

I seem awkward when addressing 
touchy subjects 
Goal Setting: 

Make sure focus on being considerate 
and not judgmental when discussing 
touchy subjects 

Poor 
83 (64%) 



Completed with care 

Specific, sufficiently detailed assessment 

Achievable, focused goals 

Goals closely linked with self-assessment 



Self-Assessment: 

I was no longer making eye contact with the 
patient when I asked about substance use 
Goal Setting: 

Think about the patient as a person when I ask 
about touchy subjects and explore my own 
feelings about substance use 
Good 
46 (37%) 
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Statistical analyses 

Frequency distributions are reported for the quality of 
students' initial SAGS. Simple Chi-square analyses were 
used to explore changes in the quality of students' SAGS 
from Time 1 to Time 2. Analyses of variance (ANOVAs) 
were used to explore the effects of both time as a repeated 
measures factor (three time points of assessment) and of 
quality of SAGS skills (as the between subjects factor, 
poor vs. good quality), as well as their interaction, 
on interviewing skills. To explore associations between 
changes in SAGS skills and subsequent interviewing 
skills, the between subjects factor included three groups: 
students whose SAGS skills were poor at both Time 1 and 
Time 2 (remained poor); students whose SAGS skills 
were poor at Time 1 but were rated as good at Time 2 
(improved); and students whose SAGS skills were rated 
as good at both Time 1 and Time 2 (remained good). Post 
hoc analyses with Bonferroni corrections were used to 
isolate significant differences. 

Results 

Self-assessment and goal setting 

Quality of initial SAGS 

For the 82% (129/157) of students who turned in work- 
sheets after their first SP case, quality of initial SAGS was 
categorized as follows: 36% (n =46) 'good', 54% (n =70) 
'fair', and 10% (n = 13) 'poor' (Table 1). For all subse- 
quent analyses, 'poor' and 'fair' ratings were grouped 
together because of the small number of worksheets rated 
as poor, resulting in a dichotomous SAGS quality of 
either 'good' or 'poor/fair'. 

Change in quality of SAGS 

The percentage of all students with 'good' SAGS im- 
proved significantly from Time 1 to Time 2 (36%, n =46/ 
129 vs. 61%, 63/104). Table 2 shows that for the 104 
students who handed in both SAGS worksheets, 40% 
(n =41) had poor quality SAGS skills at both Time 1 and 
Time 2 (remained poor); 27% (n = 28) improved from 
poor quality at Time 1 to good at Time 2 (improved); and 
34% (n =35) had good quality SAGS skills at Time 1 and 

Table 2. Change in quality of SAGS from initial to time 2 
(week 10) assessment 



Time 2 



Initial 


'Poor' SAGS 
quality 


'Good' SAGS 
quality 


Chi-Square 


'Poor' SAGS 


41 (40%) 


28 (27%) 


4.65 


quality 






p < 0.05 


'Good' SAGS 


0 (0%) 


35 (34%) 




quality 









Time 2. No student in this study declined in SAGS 
quality over the course of the year. 

SAGS and interviewing performance 

Quality of initial SAGS skills and both concurrent 
and subsequent interviewing skills 
At baseline, the quality of students' SAGS skills was 
not associated with their interviewing skills exhibited 
in the first SP exercise: students judged to have 'good' 
SAGS scored 70% well done on the OSCE (SD =23%), 
vs. 72% well done (SD = 23%) for those with 'poor' 
SAGS (t-test =0.39, p =0.70). 

Initial SAGS skills were associated with a subsequent 
change in interviewing skills over the three SP exercises 
(Week 1, Week 10, and Week 17). The main effect of time 
was significant (F =9.32, p < 0.001), with z-scores show- 
ing a general increase over time, while the quality of 
initial SAGS was not (F = 1.23, p =0.67). However, the 
interaction of these two factors was significant (F =4.16, 
p =0.044) with a pattern suggesting a latency effect 
(Fig. 1). At Time 1, interviewing skill scores were similar 
for both those with poor and good quality SAGS skills. 
At Time 2, students with poor-quality SAGS skills had 
higher mean interviewing scores than those with good 
SAGS scores. By Time 3, however, the pattern switched, 
and students with good baseline SAGS skills had sub- 
stantially higher interviewing scores than those with poor 
baseline SAGS skills. These standardized scores show 
that this statistically significant difference is a moderate 
sized effect, about 0.5 of a standard deviation (or about 
12-15% difference in % items rated as well done on our 
checklist). 

Change in quality of SAGs and subsequent 
interviewing skills 

An analysis of variance with time as the repeated measure 
factor and the change in quality of SAGS skills as the 
between subjects factor demonstrated a significant inter- 
action effect (F =2.44, p =0.047, n = 104). As shown in 
Fig. 2, the pattern of changes in interviewing skills 
between Time 2 and Time 3 was dependent on group 
membership. Students who had consistently poor SAGS 
skills (Group 1, n = 30) improved in their interviewing 
skills at Time 2 but then decreased substantially at Time 3. 
Students whose SAGS skills improved (n=31) had 
interviewing skill scores that hovered around the mean 
and did not change from Time 2 to Time 3. In addition, 
those students who had consistently good SAGS skills 
(n =25) appeared to experience, on average, a decline in 
their interviewing skills between Time 1 and Time 2 but 
then a meaningful increase from Time 2 to Time 3. 

Discussion 

We describe the evolution of the quality of medical stu- 
dents' SAGS ability and its relationship to demonstrated 
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—A— Initial Quality of SAGS Poor (n=69) -•■ Initial Quality of SAGS Good (n=35) 

• 0.31 




Initial Time 2 (Week 10) Time 3 (Week 17) 

Fig. 1. Initial Quality of SAGS and Subsequent Interviewing Skills (n = 104). 

ANOVA Quality of SAGS x Time Interaction _F = 4.16, p =0.024. Post hoc Bonferroni-corrected pairwise comparisons showed 
that for those with poor initial quality of SAGS, interviewing scores decreased while for those with good initial quality of SAGS, 
scores increased. 



interview skills over the course of the first year of medical 
school. While overall both the quality of students' SAGS 
in this domain and interviewing skills improved over time 
- hopefully a reflection of an effective curriculum - 
subsets of students based on initial and changed SAGS 
ability had very different patterns, which may have 
implications for both curriculum and assessment. 

Higher baseline SAGS skills were associated with an 
initial decline and then a substantial increase in commu- 



nication scores. This pattern may be attributable to 
a phenomenon known as expertise reversal in which 
students with more automatic and unconscious compe- 
tence temporarily do poorly on assessment strategies 
designed for novices (23). Students with solid interview- 
ing skills who then consciously assess and reflect on those 
skills may find themselves 'coming undone' for the 
next assessment but are rapidly able to re-integrate those 
skills with what they learned through self-reflection and 



■ Initial □ Time 2 (Week 10) ■ Time 3 (Week 17) 





0.21 


0.08 

■ 




0.08 






-0.02 








-0.09 




-0.28 





-0.35 

Quality of SAGS Remained Quality of SAGS Improved Quality of SAGS Remained 
'Poor' (n=30) (n=25) 'Good'(n=31) 



Fig. 2. Change in Quality of SAGS and Interviewing Skills Over Time (n =86). 

ANOVA Change in Quality of SAGS x Time Interaction _F = 2.44, p =0.044. Post hoc Bonferroni-corrected pairwise 
comparisons showed that the interviewing scores of students whose quality of SAGS rating remained poor significantly 
decreased from Time 2 to Time 3 while the interviewing scores of those whose SAGS ratings remained good increased from 
Time 2 to Time 3 (after decreasing from Time 1 to Time 2). 
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goal-setting and demonstrate improved interviewing 
skills when next assessed. Further exploration into the 
meta-cognitive, motivational and other characteristics of 
this group of students is likely to identify a highly self- 
regulated group, perhaps over 50% of students, who may 
thrive in a more self-directed curriculum. 

Low-quality SAGS' ability at baseline that remained 
poor 10-weeks later were risk factors for failing to sustain 
growth in interviewing skills. While these students' inter- 
viewing skills, on average in comparison to their peers, 
improved from Time 1 to Time 2, they declined by Time 3. 
The initial increase could be interpreted as a practice 
effect, one that could not be sustained by the end of the 
course. We need to better understand these high-risk 
students. They are likely a diverse group with respect 
to engagement in certain aspects of medical training. 
Given the importance of interviewing skills to patient 
outcomes in practice this may have significant implica- 
tions for general clinical competence. Within this group 
could also be those with poor self-regulation skills in 
general (24). One might predict that medical students are 
'super students' given the impressive academic success 
they need to demonstrate to gain admission. However, 
researchers using standard measures still identify a sur- 
prising, range of self-regulation and self-efficacy among 
enrolled medical students (25). 

Students who went from having poor SAGS skills to 
good SAGS skills did not demonstrate any change in their 
interviewing skills. Given that effects for the other groups 
differed over time in this study, we suspect that the impact 
of enhanced SAGS skills may not be discernible for some 
time and therefore that a more longitudinal design might 
have been able to document a lagged effect. For this group, 
more time and experience may be needed for their en- 
hanced SAGS skills to lead to improved practice. Further 
research might benefit from including longer time frames 
in order to capture these potentially lagged effects. 

For students who demonstrate the ability to self-assess 
and set goals with specificity, instructional strategies like 
structured self-assessed VTR with group review may be 
sufficient to ensure competence. 

We have found that some students do not acquire ade- 
quate SAGS skills through a curriculum where bench- 
marks are provided and generation of behaviorally 
specific feedback and concrete goal-setting is modeled 
and that these students are at risk of not acquiring 
communication skills on par with their peers. Future 
investigations should examine whether more explicit 
feedback on the actual SAGS skills would lead to growth 
in both areas. Students who do not demonstrate this 
skill may require more explicit training focused on their 
SAGS skills in order to stimulate growth in their com- 
munication skills. However, it is not yet clear what the 
natural course of SAGS skills is over time, nor what 
interventions are likely to be effective for these high-risk 



students. Self-regulation requires significant motivation 
and is domain specific - so some students may show the 
ability to self-monitor and achieve in a domain of great 
interest to them personally and not in others. A major 
driver of the call to better integrate foundational knowl- 
edge and clinical experiences in medical education is to 
provide students with increased motivation to learn the 
foundational knowledge by illustrating it in context of 
practice (26, 27). Ultimately, medical students in order to 
honor the tenets of medical professionalism must develop 
strategies to demonstrate competence in domains where 
they lack personal motivation (28). SAGS skills may be 
an important aspect of this professional development 
challenge. 

Our study suffers from many of the limitations asso- 
ciated with evaluating educational innovations: a small 
sample size, short time frame, and limited generalizabil- 
ity. In addition, our assessment of the quality of SAGS 
skills is somewhat simplistic and may miss important 
nuances in pattern of students' skills, for example, stu- 
dents who are reflective but not skilled in setting goals or 
students who are able to set focused, actionable goals, but 
find they are unable to follow through on those goals. 
Finally, some students failed to complete the worksheets 
and therefore are not included at all time points and may 
have affected the results. 

Conclusion 

According to the 2008 review by Colthart et al. of the 
medical education self-assessment literature, 'there is no 
solid evidence base within the health professions' litera- 
ture which establishes the effectiveness of self-assessment 
in . . . influencing learning activity' (16). Our study begins 
to build the link between goal setting and development of 
interviewing skills, demonstrating that better SAGS 
ability is associated with improvement in performance 
of interviewing skills in a standardized scenario. Students 
with poor quality self-assessment skills (40% of our 
students) did not show this improvement. More work 
needs to be done to understand the longer-term implica- 
tions of these findings, and to better understand the 
constructs being examined, both of which may have 
implications for tailoring of curriculum, remediation of 
poorly performing students and medical school admis- 
sions criteria. 
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Appendix 

School of Medicine Student Academic Portfolio: Practice of Medicine Clinical Skills Interview Videotape Review 
Video Tape Review Assignment 

Review the video using the VTR SELF EVALUATION guidelines below. Assess whether or not you demons- 
trated the specific skill and then as specifically as you can, assess and explain your performance in the text box 
provided. 

Don't be surprised if the first time you watch the clip you are surprised by your looks, gestures, or the sound of your 
voice. This is common. 

Focus on your interviewing skills using the VTR SELF EVALUATION guidelines. 

Be sure to note some things that you think you did well. Don't forget that it is easy to be critical of our own performance. 
Most interviews, even by the most seasoned clinicians, can be improved in some way! 
Share this form with your POM Seminar leader by 5 PM on DATE. 

Choose a 2-3 min segment of your interview to share in class. It can be something you thought went well or not so well. 
Just prepare to tell your colleagues why you chose this part. 
VTR SELF-EVALUATION 



Communication Skills 



Did You Do 
This? 



If did NOT do, explain what 
happened . . . specific things 
you missed or did not so well 
and why 



RELATIONSHIP 
DEVELOPMENT 



INFORMATION Managed the narrative flow Q 

GATHERING Elicited patient's response using appropriate questions [ ] 

(non-leading questions, one at a time) 

Clarified information by repeating on an ongoing basis to [ ] 
make sure he/she understood 

Allowed patient to talk without interrupting [ ] 

Communicated concern or intention to help [ ] 

Non-verbal behavior enriched communication with patient [ ] 
(e.g., eye contact, posture) 

Acknowledged patient's emotions/feelings appropriately [ ] 

Was accepting/non-judgmental of patient [ ] 

Used words patient understood and/or explained jargon [ ] 

EDUCATION & ASK: Asked questions to see what patient understood/ [ ] 
COUNSELING thought 

TELL: Provided clear explanations/information [ ] 

ASK: Asked for patient to explain what she understood [ ] 



How would you rate your overall communication skills . . .? 



1 Inadequate 



2 Marginal 



3 Adequate 



4 Very Effective 



Ineffective communication 
skills will create clinical 
problems 

□ 



Uses some communication skills 
effectively but others could create 
clinical problems 

□ 



Uses most 
communication skills 
effectively 

□ 



Uses all communication skills 
effectively, minor suggestions 
would enrich 

□ 



Overall, what went well? 
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Goal setting and interviewing skills development 



List three areas of your interviewing that you would like to improve. Be specific in describing the behaviors you observed 
that you would like to change. Describe a concrete plan to work on them (in other words, what would you like to do 
differently)? 



Interviewing Areas In Need of Improvement Concrete Plan for Improving 

1 

2 
3 



What did you learn from seeing yourself that you didn't learn right after the OSCE? 



Overall, how much did you learn from this self-evaluation? 



1 2 3 4 

Nothing Really Learned A Little Bit Learned Something Learned A Lot 

□ □ □ D 
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